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The ATLAS High Level Trigger (HLT) consists of two selection steps: the second level trigger and the event 
filter. Both will be implemented in software, running on mostly commodity hardware. Both levels have a 
coherent approach to event selection, so a common core software framework has been designed to maximize 
this coherency, while allowing sufficient flexibility to meet the different interfaces and requirements of the two 
different levels. The approach is extended further to allow the software to run in an off-line simulation and 
reconstruction environment for the purposes of development. This paper describes the architecture and high 
level design of the software. 



1. INTRODUCTION 

The Large Hadron Collider LHC, currently under 
construction at CERN and scheduled to start data- 
taking in 2007, will collide protons on protons at a 
center-of-mass energy of 14 TeV. At the design lumi- 
nosity of 10 cm - 2 s , each bunch crossing, occurring 
at 25 ns intervals, will result in 23 collisions on aver- 
age. 

The trigger of the ATLAS experiment, one of the 
multi-purpose detectors at the LHC, must be able to 
reduce the interaction rate of 10 9 Hz to below the 
maximum rate that can be processed by the off-line 
computing facilities, O(10 2 ) Hz. In addition, the AT- 
LAS trigger must be able to handle the full data vol- 
ume of a detector of the complexity and size of AT- 
LAS, with 0(1 s ) read-out channels. It is under these 
constraints that the ATLAS trigger must retain the 
capability of identifying previously undetected and 
rare physics processes. A Standard Model Higgs par- 
ticle with a mass of 120 MeV, decaying into two pho- 
tons, is for example expected to occur at a rate of 
10~ 13 of the interaction rate, the proverbial pin in the 
haystack. 

Three distinct trigger steps are foreseen. While the 
first step (Level- 1 trigger) is implemented as a hard- 
ware trigger, the second and third steps, Level-2 trig- 
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ger and Event Filter, are software triggers and are 
usually referred to as the ATLAS High Level Trigger 
(HLT). 

This article describes the current status of the high- 
level design and the implementation of the HLT event 
selection software. The validation of the software is 
on-going and aims at the imminent goal of the HLT 
Technical Design Report (TDR), due in summer 2003, 
as well as at preparing for the first stage of ATLAS 
commissioning. 

After a short overview of the ATLAS trigger, the 
general strategy of the HLT event selection software 
and its main software components are described. Sub- 
ject of this article is the part of the code that provides 
the infrastructure in which to run the HLT selection 
algorithms. Specific HLT event selection algorithms 
are not discussed. Using the same software environ- 
ment to develop and maintain (off-line) on the one 
hand and on the other hand to deploy (on-line) the 
HLT selection software is a central HLT design aim. 
This article also discusses our experience with appro- 
priating off-line code for on-line use. 



2. THE ATLAS TRIGGER 

Data at the LHC are produced with the bunch 
crossing rate of 40 MHz. The ATLAS trigger (see 
Fig. ^) has the task to reduce this rate to an output 
rate of 0(200) Hz after the Event Filter. The Level- 
1 trigger p| reduces the initial 40 MHz to less than 
75 kHz in less than 2.5 fj,s, the maximum output rate 
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Figure 1: Overview of the ATLAS trigger 



alignment information. 

Though the type of selection algorithms used by 
the Level-2 trigger and the Event Filter are different, 
the software infrastructure that the algorithms use is 
kept as similar as possible between the two trigger lev- 
els. In order to achieve a high level of flexibility, it is 
desirable to retain to some degree the possibility to 
deploy an HLT algorithm either in the Level-2 trigger 
or in the Event Filter. This article describes the com- 
mon software infrastructure that is used throughout 
the HLT. Given the much more stringent performance 
constraints of the Level-2 trigger, this approach meets 
limits that will be discussed as well. 



3. THE HIGH LEVEL TRIGGER EVENT 
SELECTION SOFTWARE 



and latency the trigger hardware can tolerate. In the 
HLT 2] , where the boundary between the two trigger 
steps is purposefully kept flexible, the Level-2 trigger 
will reduce the rate to 0(2) kHz and the Event Filter 
further to 0(200) Hz. The available average latency 
of the two steps is substantially different, with ~10 ms 
for the Level-2 trigger and ~1 s for the Event Filter. 

Central to the ATLAS trigger design is the Region- 
of-Interest (Rol) concept. The Level- 1 trigger looks 
for regions of potentially interesting activity in the 
Calorimeters and the Muon Spectrometer that may 
correspond to candidates for high px objects (electro- 
magnetic, muon, tau/hadronic and jet clusters). The 
Level-2 trigger selection uses the Rol information (its 
type, position and the px of the highest trigger thresh- 
old passed) of the Level- 1 trigger as seeds for its pro- 
cessing. This strategy keeps the amount of raw data 
to be passed to the Level-2 trigger for processing at 
only a few per cent of the full event information. 

While the raw data of the ATLAS subdetectors are 
held in readout buffers (ROB), the Level-2 trigger 
processes Rol information and raw data on a Level-2 
Linux PC farm. On each node of the Level-2 farm, a 
Processing Application executes the Level-2 event se- 
lection software. Specialized, speed-optimized Level- 
2 trigger algorithms carry out the feature extraction 
in the region-of- interest indicated by the Level- 1 trig- 
ger. While the Level- 1 trigger does not combine the 
information from different subdetectors, the Level-2 
trigger covers all subdetectors sequentially and gains 
additional information from combining their data. 

Once an event has been accepted by the Level-2 
trigger, the raw data of the full event are passed to the 
Event Builder, which in turn passes on the fully-built 
event to the Event Filter Linux PC farm. On the farm 
nodes, independent Processing Applications execute 
off-line-type selection algorithms that have access to 
the full event data, including the latest calibration and 



The HLT event selection software has four main 
components: the HLT algorithms that perform event 
reconstruction and feature extraction, the HLT Steer- 
ing that calls a certain subset of the available HLT al- 
gorithms in a certain sequence depending on the types 
of Rol received from the Level- 1 trigger, the HLT 
Data Manager that gives access to the information 
contained in the raw data, and the HLT raw Event 
Data Model that specifies the object representation of 
the raw data to be used by the HLT algorithms. This 
section describes how the four components work to- 
gether and discusses the design and current implemen- 
tation of the three components, HLT Steering, HLT 
Data Manager, HLT Event Data Model, that provide 
the infrastructure for the HLT algorithms both in the 
Level-2 trigger and in the Event Filter. 

3.1. Working principle 

The following example illustrates the working prin- 
ciple of the HLT event selection software: The Level-1 
trigger finds two isolated electromagnetic clusters with 
Pt > 20 GeV each. This is a possible signature for 
the decay Z — > e + e~ for which the ATLAS trigger ac- 
cepts an event, if both electrons are isolated and have 
a minimum px of 30 GeV. 

The HLT validates this hypothesis in a step-by-step 
process. Intermediate signatures are produced as re- 
sult of the algorithmic processing of a given step, and 
each intermediate signature is examined in order to 
be able to reject the hypothesis at the earliest possi- 
ble moment. This procedure is managed by the HLT 
Steering package. 

The validation procedure starts from the Level-1 
Rol information as seed, as shown in Fig. [5] In this 
example, there are two isolated electromagnetic Rols 
with p T > 20 GeV ("EM20i" in Fig. EJ). The HLT 
Steering calls an HLT algorithm ( "Cluster Shape" ) to 
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Figure 2: Example illustrating the working principle of the HLT selection software. The different steps follow each 
other in time. Each step is shown with the complete processing chain preceding it. Hence for step 4, the full chain of 
processing steps is shown, with time increasing from bottom to top. 



determine the cluster shape of the first seed. The clus- 
ter shape is found consistent with the electron hypoth- 
esis and the result is an electron candidate ("ecand"). 
The HLT Steering proceeds in the same way with the 
second seed and obtains a second electron candidate. 
These two electron candidates are considered as an 
intermediate signature ("ecand + ecand"). Since the 
event after this first step is still compatible with the 
Z — > e + e~ signature, the HLT Steering starts the sec- 
ond step of the validation procedure. In this second 
step, the two electron candidates provide the seeds for 
the algorithmic processing. The HLT Steering calls 
a track finding HLT algorithm ("track finding") for 
each of the two electron candidates. If for each of the 
electron candidates a track pointing to them is found, 
this second step results in two electrons that consti- 
tute another intermediate signature ("e + e"). Using, 
in the manner described above, the output of one step 
of the validation procedure as input to the next, the 
HLT Steering calls the appropriate HLT algorithms in 
the appropriate sequence to verify that the two elec- 
trons each have pr > 30 GeV and are both isolated. 
When finally, after the fourth step, the signature "e30i 
+ e30i" has been reached, the ATLAS trigger accepts 
the event as a candidate for the decay Z — > e + e~ with 
the desired electron characteristics. 

To guarantee early rejection, the HLT Steering can 
reject an event after any step during the validation 
procedure. If the first step in the example above had 



not resulted in two electron candidates with the ap- 
propriate cluster shapes, the HLT Steering would have 
stopped the validation of the specific Z decay signa- 
ture under scrutiny without passing to step 2. Note 
also that the HLT processing is not organized verti- 
cally (carry out the full sequence of algorithms to ar- 
rive first at a reconstructed electron for the first Rol 
and only then do the same for the second) but hori- 
zontally (carry out only the reconstruction foreseen in 
step x for the first and then the second seed and use 
the output as seeds for the next step). 

3.2. Steering 

In this section, the main ingredients of the HLT 
Steering package, Trigger Menus, Sequence Tables and 
Trigger Elements, are discussed briefly. For a detailed 
description, see Q- 

3.2.1 . Trigger Menu and Sequence Table 

In the example above, only one signature (hypoth- 
esis) and for this signature only one sequence of al- 
gorithms and intermediate signatures was considered. 
Different from this simple example, the ATLAS HLT 
accepts an event if it is consistent with at least one 
signature (hypothesis) in a whole catalog of possible 
ones. Also, depending on the Level- 1 Rols, there may 
be more than one possible sequence of algorithms and 
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Figure 3: Example illustrating the way Trigger Elements are used for communication between the HLT Steering and 
the HLT algorithms. Time increases from bottom to top. 



intermediate signatures per signature. But as in the 
example above, the HLT processing is organized hori- 
zontally, not vertically. This means also that the HLT 
does not examine one signature and only then the 
next. Instead, for each signature, the processing is 
broken up in single steps and the HLT Steering car- 
ries out the processing per step, not per signature. 

Each step is accompanied by one Trigger Menu and 
one Sequence Table. A Trigger Menu is a list of all 
intermediate signatures in its step for which an event 
can be accepted. The Sequence Table specifies which 
HLT algorithms are to be executed, given the seeds as 
input to the step. 

For more details, see 0. 

The HLT Steering deploys algorithms in a very spe- 
cific way. The task of an HLT algorithm is not to per- 
form a full scan for, e.g., all muon candidates in the 
event, independent of their location in the detector. 
Instead, the algorithm has to carry out a localized 
reconstruction, guided by a seed, limited to the area 
in the detector indicated by the seed. Hence, in the 
same event the HLT Steering calls the relevant algo- 
rithm once per seed, i.e. depending on the number of 
seeds possibly several times. 

3.2.2. Trigger Elements 

Communication between the HLT Steering and the 
HLT algorithms happens with the help of Trigger El- 
ements. The HLT Steering hands over the seeds to 



a HLT algorithm by means of Trigger Elements. A 
HLT algorithm returns the result of its processing to 
the HLT Steering in the form of a Trigger Element. 

A Trigger Element has a label, but does not hold 
any data content. It is related to Data Objects (Rols, 
tracks, clusters, etc.) and other Trigger Elements in 
a navigable way. The signatures in a Trigger Menu 
consist of logical AND/OR combinations of Trigger 
Elements. 

The following example (see Fig. |3J, simplified as 
compared to the example in Fig. El illustrates how the 
HLT Steering uses Trigger Elements to communicate 
with HLT algorithms. The Level- 1 trigger finds one 
isolated electromagnetic cluster with px > 20 GeV. 
Assume for the sake of the argument that the ATLAS 
trigger accepts events with at least one isolated elec- 
tron with pt > 30 GeV. Then the Rol found by the 
Level- 1 trigger is a possible signature for an event of 
this type. 

In step 1, the electromagnetic Rol found by the 
Level- 1 trigger is the seed for the HLT cluster shape 
algorithm. The HLT Steering creates a Trigger Ele- 
ment with label "EM20i" that has a navigable "uses" 
relationship to the Level-1 Rol as Data Object. The 
output of the HLT cluster shape algorithm is an elec- 
tron candidate that the algorithm hands back to the 
HLT Steering as a new Trigger Element with label 
"ecand" . This Trigger Element has a navigable "uses" 
relationship to the calorimeter cluster that was the re- 
sult of the algorithmic processing. Since the HLT clus- 
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Figure 4: Overview of where in the ATLAS trigger raw data are sent to the HLT in bytestream format. 



ter shape algorithm was seeded by the Level- 1 Rol, 
its corresponding output Trigger Element, "ecand", 
has a navigable "seeded by" relationship with its in- 
put Trigger Element, "EM20i". Navigable "uses" re- 
lationships are also possible between Data Objects. 
This allows the HLT track finding algorithm in step 
3 to access the calorimeter cluster on which its seed 
is based, and to access the calorimeter cells on which 
the calorimeter cluster is based. In addition, via the 
"seeded by" relationship between its input Trigger El- 
ement and the input Trigger Element of the preceding 
step, the algorithm has also access to the underlying 
Data Objects of that previous step. 

In the design, the Trigger Elements represent the 
intermediate hypotheses that correspond to the inter- 
mediate signatures that the HLT Steering examines 
to decide whether to continue with the next step of 
the algorithmic processing. The Data Objects, on 
the other hand, represent the information support- 
ing these hypotheses. Differentiating between Trigger 
Elements and Data Objects has the advantage of dif- 
ferentiating clearly between the HLT Steering realm, 
containing the hypotheses, and the HLT algorithm re- 
construction realm, containing the supporting infor- 
mation. 



trigger decision, the Level-2 trigger receives the Level- 
1 trigger result, in particular the Rols, by means of a 
bytestream. In addition, the Level-2 trigger can de- 
mand the bytestream fragment from any of the ~1600 
ROBs that each hold the raw data of a part of the 
ATLAS detector. When the Level-2 trigger accepts 
an event, the Event Builder receives the Level-2 trig- 
ger result and the bytestream from all ^1600 ROBs 
and it passes the fully-built event by way of another 
bytestream to the Event Filter. 

3.3.1. Raw Data Objects and Detector Elements 

In order to allow the HLT algorithms access to the 
raw data information in these bytestreams, this infor- 
mation has to be represented in object form, as Raw 
Data Objects (RDOs). The HLT algorithms access 
the raw data information by means of the RDOs and 
the transient event store, where the RDOs are stored. 

The Raw Event Data Model (Raw EDM) is part of 
the HLT EDM and specifies the design of the RDOs 
for each ATLAS subdetector and each bytestream in 
the Data Flow system. The Raw EDM has to map 
onto one another data of very different granularity. 
The Data Flow system provides data with a minimum 



3.3. Raw Event Data Model 

Unlike the typical off-line situation where the data 
are stored in a convenient format, e.g. objects in a file 
or database, the HLT event selection software running 
on the HLT Linux farms can access data only in their 
raw format and needs to invoke the ATLAS Data Flow 
system |E IE @ m order to do so. 

The Data Flow system transfers raw data to the 
HLT in bytestream format. Figure 0] illustrates where 
in the ATLAS trigger system raw data is exchanged 
via bytestreams. In the event of a positive Level- 1 




DetectorElement 



Figure 5: Definition of a DetectorElement for the 
example of the ATLAS Muon Spectrometer. 
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granularity of one ROB, while a HLT algorithm typi- 
cally needs data organized by geometric detector unit. 

For example, in the ATLAS Muon Spectrometer, a 
geometric detector unit of interest for a HLT selec- 
tion algorithm is a chamber. A chamber is a physical 
unit, for example of Monitored Drift Tubes (MDT), 
that share the same physical support structure, are 
installed as a whole and may be described by a single 
set of condition (e.g. alignment, etc.) constants. The 
Raw EDM specifics for all ATLAS subdetectors the 
object representation of the relevant geometric detec- 
tor units, the Detector Elements (see Fig. |SJ. 

In the case of the MDT chambers of the Muon 
Spectrometer, the Detector Element corresponds to 
a chamber. In the same way as a chamber can be 
viewed as a collection of readout channels, a Detector 
Element object is a collection of RDO objects which 
each hold the information of one readout channel. 

RDOs are stored in the transient event store ordered 
according to Detector Elements. The transient store 
contains for each subdetector a container of Detector 
Elements (see Fig. 0. Each Detector Element has 
a unique identifier [3| by which it can be retrieved 
from the store, thus allowing with one request to the 
transient store to retrieve collections of RDOs, which 
respresent collections of readout channels in geometric 
detector units. 



3.4. Bytestream converters 



sponsible for storing the RDOs in the correct Detector 
Element order in the transient store. 

Raw data access via Bytestream Converters pro- 
ceeds in the following way (see Fig. (JJ. The HLT 
Steering calls an HLT algorithm by passing to it a 
Trigger Element with a navigable link to a Rol Data 
Object that contains the Rol position. This Rol can, 
for example, be located in the Muon Spectrometer. 
The HLT algorithm typically needs access only to 
the raw data in those Muon Spectrometer chambers 
within a certain region around the Rol position (see 
Fig- El- These chambers correspond to a certain set 
of Detector Elements. The task of identifying this 
set of Detector Elements, given a region in the pseu- 
dorapidity n and the polar angle <f>, is performed by 
the Region Selector hrfl. The Re gion Selector returns 
to the calling HLT algorithm the list of identifiers of 
the appropriate set of Detector Elements. The HLT 
algorithm requests the Detector Elements from the 
transient store with the help of this identifier list. 

At this stage, two situations can occur. If the re- 
quested Detector Elements are already available in the 
transient store, the store simply returns them. If they 
are not available, however, the corresponding raw data 
information first has to be requested from the Data 
Flow system. In this case, the transient event store 
invokes the correct Bytestream Converter, which in 
turn requests the raw data in bytestream format from 
the appropriate ROBs. From the succession of 32-bit 
words sent by each ROB, the Bytestream Converter 
extracts the relevant information and uses it to ini- 
tialize the collections of RDOs. The thus obtained 
collections of RDOs, or Detetcor Elements, are stored 
in the transient store, which passes them on to the 
HLT algorithm. 

The extraction of raw data information into RDOs 
can be supplemented by additional data preparation 
algorithms, e.g. cluster finders for the Silicon Track- 
ers. The object representation of extracted and pre- 
pared raw data are called Reconstruction Input Ob- 
jects. Their definition is part of the HLT EDM. They 
are organized inside the transient store in collections 
as are RDOs, and can be requested by an HLT algo- 
rithm in the same way as RDOs. 



The raw data in the bytestream format in which 
they are held by the ROBs can be viewed as a repre- 
sentation of the raw data information in an event. The 
RDOs, organized by Detector Element, are another 
representation of this information. The conversion 
from bytestream representation to RDO representa- 
tion is the task of Bytestream Converters. They carry 
out the conversion on demand of an HLT algorithm. 
Their implementation consists of a general infrastruc- 
ture part that uses subdetector specific methods for 
decoding the information from bytestream format to 
RDO format. The general part is, for example, re- 



4. OFF-LINE CODE IN ON-LINE 
CONTEXT 

A high level of integration of the code designed for 
off-line use and the code designed for on-line use is 
clearly a desirable design goal. Ideally, it should be 
possible to develop and maintain on-line software on 
simulated and real data in a well-tested, flexible and 
user-friendly environment as is offered by the off-line 
software. This does not only apply to the HLT algo- 
rithms, but also to the HLT infrastructure code used 
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Figure 7: HLT algorithm access to raw data information by means of Bytestream Converters and the transient event 
store. 
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Figure 8: Example illustrating the task of the Region 



by them. Furthermore, it clearly enhances the HLT 
flexibility in dealing with changes in the LHC run- 
ning conditions when it is possible to migrate software 
freely between off-line environment and Event Filter 
and between Event Filter and Level-2 trigger. 

In the previous sections, the common software in- 
frastructure of Level-2 trigger software and Event Fil- 
ter software has been described. In order to use this 
HLT software infrastructure and the HLT algorithms 
in an off-line environment, the HLT software has to 
comply with the basic design principles of the ATLAS 
off-line framework. 

The ATLAS off-line framework, Athena [HI, is 
based on a GAUDI 12| (developed by the LHCb 
experiment) core. As a consequence, Athena has 
adopted a number of the basic principles of the 
GAUDI design. Examples are the separation of the 



data from the algorithms producing and consuming 
the data, separation of the transient and the persis- 
tent representation of data and the use of converters 
that convert one data representation into another. 

For the HLT event selection software this means, for 
example, that HLT algorithms are forced to commu- 
nicate indirectly, via the transient event store. This 
requirement is met by the use of Trigger Elements, 
as described in Sec. 13.2.21 It also means that per- 
sistent data can be made available to the HLT event 
selection software only by means of converters that 
convert them into a transient representation that is 
then stored in the transient event store, from where 
HLT algorithms can request data. The Bytestream 
Converters discussed in Sec. 13. 41 and their use as illus- 
trated in Fig. comply with this requirement. Fur- 
thermore, using the HLT event selection software in 
Athena implies at the very least common interfaces to 
general services, and may even mean using the very 
same services, e.g. database access tools, geometry 
service etc., by Athena and by the on-line framework. 

The HLT event selection software also needs to 
comply with the much more stringent on-line perfor- 
mance requirements concerning speed and robustness. 
Hence, also any off-line-imposed elements or code el- 
ements imported from the off-line environment must 
meet the on-line performance requirements. They are 
considerably more stringent than would be otherwise 
needed for pure off-line use. 

The following section describes the current off-line 
dependencies introduced into the HLT software for the 
above mentioned reasons. The experience gained so 
far in the attempt of using off-line code in the on-line 
context of the trigger is discussed. 
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Figure 9: The four components of the HLT selection software and their dependencies on off-line software. 



4.1. Current Off-Line Dependencies 

A high level of integration between off-line and on- 
line code can be achieved in different ways. One pos- 
sibility consists in choosing the interfaces in the off- 
line and in the on-line code such that on-line software 
can be used in the off-line framework and vice versa. 
Another possibility is to use the same framework for 
off-line and on-line purposes. In the concrete ATLAS 
situation, this means using Athena as on-line frame- 
work as well. In addition, any combination of these 
two extremes is possible. 

In the current implementation of the HLT soft- 
ware, Athena is used as framework in the Event Fil- 
ter. Level-2 employs a specialized framework that 
uses interfaces compatible with those in Athena, so 
that any Level-2 infrastructure and algorithm code 
can also be used within Athena (see next section). The 
HLT infrastructure code (e.g. transient event store, 
Bytestream Converters, etc.) currently re-uses code 
developed in Athena. 

Figure |5| depicts the current off-line dependencies 
in the HLT event selection software. The HLT frame- 
work depends on the off-line framework. For the 
Event Filter this means re-use of Athena as frame- 
work, for the Level-2 trigger this means use of a 
slightly modified version (see next section) of the 
Athena core software, based on GAUDL The Data 
Manager in the current implementation is the off-line 
transient event store, StoreGate [l|. The HLT EDM 



re-uses the off-line EDM. In the Event Filter, off-line 
algorithms are re-used as selection algorithms. In the 
Level-2 trigger, specialized Level-2 algorithms are em- 
ployed which, however, are set up such that they can 
also be run in the Athena framework. 

The execution of the HLT event selection software 
is controlled by a Processing Application that is part 
of the HLT Data Flow and Data Acquisition software. 

4.2. The Special Level-2 Environment 

The performance constraints imposed by the HLT 
are most stringent for the Level-2 trigger. There, the 
average latency available to the HLT selection soft- 
ware for accepting or rejecting an event is ~10 ms. 
The Level-2 software needs to be able to handle a 
data input rate of up to 75 kHz and to sustain a data 
output rate of 0(2) kHz error-free and deadtime-free 
for extended periods of time. Requirements of such 
stringency with respect to speed and robustness are 
usually not met by software designed for pure off-line 
use. An additional complication arises from the need 
for multi-threading in the Level-2 trigger. 

In order to keep the idle time of the CPUs in the 
Level-2 Linux PC farm to a minimum, each CPU pro- 
cesses in parallel three different events, i.e. each Level- 
2 CPU carries out at the same time three execute 
loops. This is in marked difference to the situation 
foreseen in Athena, where the initialization phase is 



WEPT004 



CHEP03 - Conference for Computing in High Energy and Nuclear Physics, La Jolla, CA, USA, March 2003 9 



Athena 



Level2 



Initialize 



Execute 

~~T~ 



Initialize 



Finalize 



Finalize 



Figure 10: Difference between the off-line framework 
Athena and the framwork needed for the Level-2 trigger, 
where multi-threading is required. 



followed by a single execute loop that ends in a fi- 
nalization phase (see Fig. I10[l . In the case of run- 
ning inside Athena, Athena controls the event execute 
loop; in the multi-threaded Level-2 software the con- 
trol needs to be with the HLT Data Flow software. In 
addition, in the Level-2 software, all algorithms and 
services have to be configured and initialized strictly 
outside of the event execute loop. Hence Athena can- 
not be used as HLT framework for the Level-2 trigger. 

In order to comply nonetheless with the design 
goal that the Level-2 event selection software be us- 
able with the Athena framework for development and 
maintenance purposes, an interface layer between the 
HLT selection and the HLT Data Flow software was 
implemented, the so-called Steering Controller 
The Steering Controller uses only a minimal set of 
GAUDI features. In addition, it uses a modifica- 
tion to the GAUDI base libraries that allows thread- 
specific instantiation. The standard GAUDI com- 
mand EventLoopM gr — ► executeEventQ is, for ex- 
ample, replaced by EventLoopM gr_(threadl D) — > 
executeEventQ, where threadID corresponds to 
the number of the thread (1, 2, 3). Accord- 
ingly, GAUDI services are called with the com- 
mand AnyServiceNeededByAlgo_(threadLD) and al- 
gorithms with AnyHLTALgorithm_(threadLD), etc. 
The possibility of thread-specific instantiation has 
since been included in GAUDI as a standard feature. 
The Steering Controller can be used within the off- 
line framework Athena because in a single-thread en- 
vironment the _(threadLD) suffix collapses to a blank. 
Beyond this the Steering Controller uses only stan- 
dard GAUDI features, which are all contained in the 
GAUDI-based Athena. 

The special Level-2 feature of running in multi- 
thread mode may cause problems when using stan- 
dard software tools as, e.g., the Standard Template Li- 
brary (STL), even when the software components used 
are in principle thread-safe. An unexpected prob- 
lem of this kind arose with the specific implementa- 
tion of STL containers in use in the HLT code. The 
container implementation, though thread-safe, proved 
highly thread-inefficient by setting unnecessary locks. 
The problem was overcome by modifying by hand the 
container allocator implementation |l5| . 

It should be noted that the problems arising from 



the use of multiple threads in the Level-2 trigger soft- 
ware are not present for the Event Filter. The Event 
Filter uses multiple threads, but the Data Flow and 
HLT event selection software are set up in a different 
way. As a result, it is possible to use Athena as is as 
the framework for running HLT selection algorithms 
in the Event Filter ^||. However, when Athena was 
used for the first time in the Event Filter, it became 
aparent that an inacceptable 30% of the total process- 
ing time per event was spent in the off-line Raw EDM 
available at that time. This problem has since been 
remedied, but illustrates the need for a thorough eval- 
uation of the suitability of off-line code for use in the 
HLT software. It also illustrates the need for a close 
and continuous collaboration between the HLT ( "on- 
line" ) and the "off-line" communities in order to adopt 
successfully "off-line" software for "on-line" use. 



5. CONCLUSIONS AND OUTLOOK 

The architecture of the HLT selection software de- 
scribed in this article is currently being validated 
by implementing the software for full vertical trigger 
slices. Two vertical slices are currently under develop- 
ment, an electron/gamma slice and a muon slice. The 
vertical trigger slices comprise the software to simu- 
late the full chain of trigger decisions, starting with 
the Level-1 trigger, that leads to identifying an event 
as containing a single electron, gamma or muon. Inte- 
gration of the HLT selection software with the other 
components of the HLT software is on-going with the 
goal of running the electron/gamma and muon verti- 
cal slices in testbeds of the foreseen Linux PC farms. 
Detailed measurements of timing and physics perfor- 
mance of each part of the HLT software are under way 
and will be reported in the HLT Technical Design Re- 
port in summer 2003. 

A central goal of the current implementation of the 
HLT selection software is a high level of integration be- 
tween off-line and on-line code. This goal has sparked 
off a very fruitful collaboration between the HLT and 
the on-line software developer communities. The at- 
tempt of adapting and re-using off-line code in the 
HLT software has been carried furthest in the specific 
way of accessing raw data in the HLT by means of 
Bytestream Converters that are hidden behind a call 
to the off-line transient event store, StoreGate. 

Given the ever increasing available CPU speed, al- 
lowing an ever higher level of abstraction, the concept 
of re-using off-line code in the on-line context appears 
a logical development. However, given the stringent 
constraints and performance requirements of the HLT, 
the present implementation of the HLT selection soft- 
ware and the heavy re-use it makes of off-line software 
is clearly experimental. This is in particular true for 
the Level-2 trigger. The results of the on-going valida- 
tion effort will show if the chosen ansatz is sustainable. 
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